Automatic Identification of Non-compositional Phrases

Non-compositional expressions present a special challenge to NLP applications. We present a method for automatic identification of non-compositional expressions using their statistical properties in a text corpus. Our method is based on the hypothesis that when a phrase is non-composition, its mutual information differs significantly from the mutual informations of phrases obtained by substituting one of the word in the phrase with a similar word. 1 I n t r o d u c t i o n Non-compositional expressions present a special challenge to NLP applications. In machine translation, word-for-word translation of non-compositional expressions can result in very misleading (sometimes laughable) translations. In information retrieval, expansion of words in a non-compositional expression can lead to dramatic decrease in precision without any gain in recall. Less obviously, non-compositional expressions need to be treated differently than other phrases in many statistical or corpus-based NLP methods. For example, an underlying assumption in some word sense disambiguation systems, e.g., (Dagan and Itai, 1994; Li et al., 1995; Lin, 1997), is that if two words occurred in the same context, they are probably similar. Suppose we want to determine the intended meaning of "product" in "hot product". We can find other words that are also modified by "hot" (e.g., "hot car") and then choose the meaning of "product" that is most similar to meanings of these words. However, this method fails when non-compositional expressions are involved. For instance, using the same algorithm to determine the meaning of "line" in "hot line", the words "product", "merchandise", "car", etc., would lead the algorithm to choose the "line of product" sense of "line". We present a method for automatic identification of non-compositional expressions using their statistical properties in a text corpus. The intuitive idea behind the method is that the metaphorical usage of a non-compositional expression causes it to have a different distributional characteristic than expressions that are similar to its literal meaning. 2 I n p u t D a t a The input to our algorithm is a collocation database and a thesaurus. We briefly describe the process of obtaining this input. More details about the construction of the collocation database and the thesaurus can be found in (Lin, 1998). We parsed a 125-million word newspaper corpus with Minipar, 1 a descendent of Principar (Lin, 1993; Lin, 1994), and extracted dependency relationships from the parsed corpus. A dependency relationship is a triple: (head type modif ie r ) , where head and modif ier are words in the input sentence and type is the type of the dependency relation. For example, (la) is an example dependency tree and the set of dependency triples extracted from (la) are shown in (lb).